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I. INTRODUCTION 



Advances in nanotechnology permit nowadays to implement feedback control actions on 
nanodevices. For instance, in surface topography, the deflection of a cantilever is captured by 
a photodetector that records the angle of reflection from a laser beam focused on the mirrored 
surface on back side of the cantilever. Position feedback control is used to maintain the probe 
at a constant force or distance from the object surface. Position can also be differentiated 
allowing to apply a velocity dependent external force. A velocity dependent feedback control 
(VFC) has been recently implemented to reduce thermal noise of a cantilever in atomic force 
microscopy (AFM) [26] and in dynamic force microscopy [44]. A contribution to a rigorous 
thermodynamical foundation of macromolecules under (VFC) operating in nonequilibrium 
steady state is provided in [21I]. The entropy production rate is there decomposed into a 
positive entropy production rate (PEPR) and an entropy pumping rate (EPuR). The latter 
indicates how much entropy is pumped out or into the macromolecule by the control force. It 
may render the overall entropy production negative. This is at the basis of a macromolecular 



cooling mechanism 6 



26 



44|. 



In this paper, we study entropy production in the presence of an external force in a more 
general situation. Our approach is new and complementary to 2l|] in that we study the free 



energy change rather than the total entropy change of the heat bath and of the Brownian 
particles as done in 2l{]. As "distance" between two probability densities associated to the 
unperturbed and perturbed evolution we employ the information relative entropy (in the 
quantum case, the von Neumann relative entropy for density operators). We show that it 
is possible to derive some basic formulas on the entropy production rate that extend those 



of [2l|] whenever the evolution of the physical system is in some suitable sense Markovian. 
We study nonequilibrium thermodynamical systems with finitely many degrees of freedom. 
Corresponding results are also sketched for n- level closed and open quantum systems. The 
perturbation of the Hamiltonian is interpreted as a control function which is designed by 
the controller in order to obtain a desired behavior of the system (reduction of thermal 
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noise, transfer to another state, etc.). Among potential applications, we mention molecular 



kinetics 



macromolecular cooling 
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44| . quantum computation [3J]. As it is 



well-known, relative entropy plays a central role in many areas of modern science besides 



physics such as mathematical statistics, information theory, probabi 



and quantum information processing, see e.g. 



, ,9 
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ity, signal 
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and references therein. Some of these results have been announced without proofs in our 



conference papers 
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The paper is outlined as follows. In the next section, we consider finite dimensional, 
nonequilibrium thermodynamical systems. In Section III, we derive a basic formula on 
relative entropy evolution for probability densities satisfying a continuity type equation. 
This result is then applied in the following section to controlled thermodynamic systems. 
Section V is devoted to the study of von Neumann entropy production for closed and open 
finite-dimensional quantum systems. In the Appendix, we show that the basic result may 
also be extended to non-Markovian finite-energy diffusions. 



II. THERMODYNAMIC SYSTEMS 



Consider an open thermodynamic system whose macroscopic evolution is modelled by 
an n-dimensional Markov diffusion process {x(t);to ^ The components of x form a 
complete set , i.e. all other variables have a much shorter relaxation time [l3]. Let p{x) be 
the Maxwell-Boltzmann probability density corresponding to thermodynamical equilibrium 



p{x) = Z ^ exp[- 



kT 



(II.l) 



Here H is the (continuously differentiable) Hamiltonian function, and the (forward) Ito 
differential of x is 



dx{t) 



1 

2¥T 



T.lfVH{x{t))+u{x{t),t) 



dt + T^dW, 



(11.2) 
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where W is a. standard n-dimensional Wiener process. The probabihty density pt of x{t) 
satisfies the Fokker-Planck equation 



dt 



2kT 



WVH + u]p 



1 ^ 
2 



In the uncontrolled case m = 0, under reasonable assumptions, see e.g. 



(11.3) 



as t cxD, the 



density pt of x{t) tends to p in relative entropy and, consequently, in total variation 24 1. 



For the ergodic properties of this class of diffusions see e.g. 



43l . Section 7.5]. 



Example II. 1 In polymer dynamics 13], the macromolecule is described by a Hamiltonian 



H{x,y) = -{y,My) + ^{x), 



where M stands for the direct sum 



M = Mi 



Mn, 



3) 



fc = l,...,iV, 



Here x and y are 3A^-dimensional vectors, with Xi, yi 3-dimensional position and momentum 
of ith hard building block of macromolecule. Moreover, ip{x) is the internal potential of 



macromolecule (in AFM experiment 



261], (p{x) = Kx'^/2, where K is the spring constant 



of cantilever). Random collisions between solvent water molecules and building blocks of 
macromolecule are modeled by the formal derivative of a Wiener process, namely Gaussian 
white noise. The six dimensional stochastic process {qi^x, qi,y, Qi,z,Pi,x,Pi,y,Pi,z) associated the 
ith block obeys the equation 



dqia = dp^^H{q,p)dt, 

dpia = [-dq^^H{q,p) + fia + Uia{q,p)] dt + Tll^dWjp{t), 



(11.4) 
(11.5) 



where Einstein's convention has been used. Here / is a frictional force, u a position-velocity 
dependent control. In the AFM experiment, / = —'jV, u = —aV, 7 > 0, a > 0, with 
V a velocity. The control here acts like a frictional force on the macromolecule. Since 
the frictional coefficient has been increased, one can introduce an effective temperature Tgfj 
which is lower than the thermostat temperature T. As is well-known, different uncontrolled 
{u = 0) versions of this model 19|] play an important role also in other applications such as 



nonlinear circuits with noisy resistors 



45]. 



To simplify the writing, we shall assume henceforth that in (111.2^ SS-^ = cr^/n- The results 



of this paper, however, extend in a straightforward way to the case where the diffusion matrix 
SS"^ is any symmetric, non negative definite (possibly singular as in (111.40 - 011. 5p ) matrix. 
Let us first recall a few basic concepts concerning the uncontrolled, nonequilibrium system 
(III. 2p . Let us introduce the fluxes J{x,t) and forces $(a;,t) by 



J{x,t) = -^a^Vptix) - ^a^VH{x)pt{x) 
t) = -V/i(x, t), 

where p = H + kT log pt is the electrochemical potential. Notice the following: 



1. The Fokker-Plank equation (III. 31) may be rewritten (see e.g. 17i]) as a continuity 
equation 

^ + V-J = 0. (II.6) 

2. Both fluxes and forces are zero in equilibrium. Moreover, 

2 

J{x,t) = ^^x,t)pt{x), (n.7) 

which plays the role of constitutive relations. 

3. For p and a nonnegative measurable functions on M", we define the information relative 
entropy [divergence, Kullback-Leibler distance) by 



As it is well known 



/ log— pdx. 



22l |. when p and a are integrable functions with 
p{x)dx = / „ a{x)dx, 



we have ©(p||cr) > 0. Moreover, D(p||cr) = if and only if p = cr. Define the free 
energy functional 

Fipt) = kT f \ogP^ptdx = kTB{pt\\p). 

Jm. p I — I 

The free energy decay may now be expressed as [l7| 

j/{Pt) = iVlog^lVrfx = - / Jix,tMx,t)dx. (n.8) 



Suppose now that, like in the AFM experiment, the thermodynamic system is subject to 
a feedback control action so that the macroscopic evolution is given by fill. 21) with m 7^ 0. 
The density p" of the solution x" satisfies the controlled Fokker-Planck equation flll.Sp . We 
are interested in the evolution of D(p"||p°), where {p^^t > to} is an uncontrolled evolution 
(u = 0) . We first need a simple but useful result. 

III. A RELATIVE ENTROPY PRODUCTION FORMULA 

Consider two families of nonnegative functions on M"" : {p^; < t < ti\ and {p^; < t < 
ti}. We are interested in how the relative entropy 3{pt\\pt) evolves in time. 
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Assumptions: 



Al There exist measurable functions f{x,t) and f{x,t) such that {pt',tQ < t < ti} 
and {pt] to <t < ti} are everywhere positive solutions of 

dpt 



g,+V.(/p,) = 0. 

t + v.,/>,)^o. 



(III.9) 
^III.IO) 



A2 



For every t G [to,ti] 



lim f{x,t)pt{x) = 0, 
lim f{x,t)pt{x) = 0, 



lim f{x,t)pt{x)\og—{x) = 0. 
\x\^oo p^ 

Theorem III.l Suppose'S]){pt\\pt) < oo,Vt > 0. Assume moreover A\ and A2 above. Then 



Proof. 



jB{pt\\pt) = J-^n^ [(log Pi - log Pt)pt] dx 

1 dpt 1 (9pt 



'tllPtJ 



Vlog^ ■(/-/) 

Pt 



Pt dx. 



pt dt Pt dt 



Pt + \og — — ) dx 
Pt dt I 



-V ■ CfPt) + - V ■ {fpt) - log ^ V • UPt] 
Pt Pt 



dx 



V7 1 Pt 7-~ ^Pt r.Pt ~ 

Vlog — • fpt - V— ■ f—pt 

Pt Pt pt 



dx 



Pt 



Vlog^ ■(/-/) 

Pt 



Ptdx, 



where we have used (IIII.9l) -( llII.10l) and integration by parts (the boundary terms are zero 
because of Assumption 2). Q.E.D. 

For pt{x) = 1, we have that — D(pj||pt) = S{pt) the entropy. Taking f{x,t) = 0, we see that 
the first condition in A2 is verified. Theorem IIII.ll then gives (exchanging pt with pt): 
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Corollary III. 2 Suppose {pf, to < t < ti} is of class and 



lim v{x,t)pt{x) = 0, lim v{x, t)pt{x) log pt{x) = 0. 



Suppose S{pt) < oo,Vt G [to7^i]- Then 



d 
di 



SiPt) = [Vlogp, ■ f]ptdx. (III.ll) 



IV. ENTROPY PRODUCTION FOR CONTROLLED EVOLUTION 

Consider now again the controlled thermodynamic system of Section 11 flll.2l) . Let 
denote the density of the controlled process satisfying (]11.3|) . We are interested in the 



evolution of D(p"||p°), where {pt,t > to} is an uncontrolled evolution {u = 0). First of all, 
recall that the Fokker-Planck equations of the uncontrolled and controlled system may be 
written as continuity equations as in (111.61) Thus, we can apply Theorem 1111.11 with 

2 2 2 2 

/ = -^^H{x) - y VlogpO(x), / = -^VH{x) + u{x,t) - y Vlogpr(x). 



We get 



dt 



Suppose now p° = p, where p is the Maxwell-Boltzmann distribution flll.ip . We get 
Theorem IV. 1 Under assumptions Al and A2, 

>(p:'II?) = 4 /m- iiviogf viogf .«pr<ix. (IV.13) 

Remark IV. 2 Formula (11V. 131) generalizes the decomposition of the entropy production 
exhibited in ^] for the controlled Langevin equations. In 21j, the total entropy change of 
the heat bath and of the Brownian macromolecules is studied. The entropy production rate 
(EPR) is decomposed into the sum of two terms. The first, named PEPR {positive entropy 
production rate) , is an always positive term expressed as the product of the thermodynamic 
force and the corresponding flux as in (111.8|) . The second, named EPuR {entropy pumping 
rate), describes the amount of entropy pumped out of or into the macromolecule by the 



external agent. We recognize that flIV.131) implies that — ^D(pJ'||p) is also decomposed into 
an always positive term and into a term depending explicitly on the control function. 



One can try to employ flIV.13|) to analize macromolecular cooling 



m 



26 



4J]. Another 



direction of application is the following. Suppose we are interesting in modifying the rate 
at which the solution pt of fill. 31) tends to the invariant density flll.l|) . Let 

and consider in fill. 20 the feedback control 

u{x,t) = -a{t)V log ^{x). (IV.14) 

P 

Then, satisfies the Fokker-Planck equation 

^ - V . ((^ + "W) ^^^P") = (y + AP". (IV.15) 

A few observations are now in order. 

1. Although the feedback control law is nonlinear in pj*, equation f llV.lSp is linear, 

2. the initial value problem for equation f lIV.151) is well posed since ^ + a{t) > 0; 



3. equation fllV.lSp still has as invariant density the Maxwell-Boltzmann distribution 

dnn); 



4. it is conceivable to solve f]IV.15|) off-line, and consequently compute the feedback law 
fllV.MD beforehand. 



5. the flow of one dimensional probability densities {p";t > 0} of x^{t) satisfying flll.2p 



with the control given by flIV.14p is the same as for the uncontrolled stochastic process 
^ with differential 

^2 , \ 1 



d^ = ~l — + a{t) —VHiOdt + ^a^ + 2a{t)dW, (IV.16) 



provided .^(O) is distributed according to pg. 



6. the friction and diffusion coefficients in (lIV.lGP ^lthough time- varying, still satisfy 



Einstein fluctuation-dissipation relation, see e.g 



3l| 



We now employ fllV.lSP to compute the relative entropy derivative. We get 



di 



D(priip) 



a 

y 



^ J^jVlog^l'p'ldx. (IV.17) 



Hence, the controlled diffusion still tends to the Maxwell-Boltzmann distribution but at a 
different, "modulated" rate. In the linear Gauss-Markov case (i.e. when H is quadratic), 
the results assume a very concrete form. In particular, equation (IIV.ISP may be replaced by 
a linear matricial equation, see 361] for details. 



The results of this section may be readily extended to non-Markovian, finite-energy dif- 



fusions employing the Nelson-Follmer kinematics |13l . I32[ |. see the Appendix. Notice that 
this family family plays a central role in several branches of mathematical physics, see e.g. 
[l^ . The results also extend without too much difficulty to a large class of diffusions 
with constant but singular diffusion coefficient such as in the case of the Orstein-Uhlenbeck 



model of physical Brownian motion 3l|] or, more generally, in the case of model ( 111.41) - f llLSl) . 



They may also be established for a lar ge c lass of Markovian diffusion processes with local 



diffusion coefficient given the results in 



18 



28|, 



29 



30 



3l|. 



V. n-LEVEL QUANTUM SYSTEMS 



It is apparent that Theorem IIII.ll can be applied to statistical mixtures in classical me- 



chanics [35!, Section IV]. Indeed, Liouville's equation, expressing conservation of density in 
phase space, is just a continuity equation for the Hamiltonian evolution. One then gets the 
idea that it might be possible to establish a similar result in the quantum case, replacing the 
Liouville equation with the Landau-von Neumann equation for the density operator. First 
of all, we need to recall the basic formalism of statistical quantum mechanics. 
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A. Closed quantum systems 



As in standard quantum mechanics [39|], to every physical system S is associated a com- 
plex Hilbert space Hs- In the standard formulation, the state of the system is described by a 
unit vector ip ^Tis- For the sake of simplicity, here we will consider only finite dimensional 
Hilbert spaces, but results hold in the general case. 

We consider situations in which uncertainty on the system state affects our model. The 
quantum analogue of a classical probability density is a density operators p in Tis: A density 
operator is a a positive semi-definite, unit trace operator on Tis- They form a convex set 
'DiTis) and the extremals of 'DiTis) are the one dimensional orthogonal projections. These 
are called pure states, and are equivalent to unit vectors in Tis up to an overall phase factor, 
by setting p = {ip, Physical observables are represented by Hermitian operators on Tig. 

Let A be an observable: The expected value of A for a system described by a density 
operator p is defined as: 

< a >p:= trace (pA). (V.18) 
Hence, the variance for an observable A given p is naturally defined as 

Var(A)p :=< (A- < a >pY >p . (V.19) 

It is easy to see that if pp is a pure state, then exists an observable A such that the vari- 
ance Var(A)pp = 0, clarifying the definition and the analogy with the classical case. The 
time evolution for the density operator of an isolated quantum system is determined by 
the Hamiltonian, i.e. the energy observable. The dynamical equation is the Landau- von 
Neumann equation: 

thj^Pt = [H,pt], (V.20) 

where [■, ■] denotes the commutator 

[A,B] := AB-BA, 
and h is Planck's constant divided by 2n. 

11 



Quantum analogues of entropic functionals have been considered since the very beginning 



of the mathematical foundation of quantum mechanics 47| . Recently renewed interest came 



from Quantum Information applications 
entropy, that is defined as: 



34( 1 ■ We are interested here in the quantum relative 



KpIIp) := trace (p(logp - logp)). (V.21) 



We define OlogO = 0. As in the c 



of a pseudo-distance (see e.g. 



34 



assical case, quantum relative entropy has the property 



40(1). We now consider the effect of a perturbation AH 
on the evolution of quantum system originally driven by a free Hamiltonian H (we denote 
hj H = H + AH the perturbed Hamiltonian) . 

Proposition V.l Let pt and pt he the solution of k V. 20\) corresponding to the unperturbed 
and the perturbed Hamiltonians, respectively. The relative entropy production for the per- 
turbed evolution is given by: 

|d(p||p) ='-< [AH, logp] >, . (V.22) 
Proof. Observing that [p, log p] = and, consequently, 

—trace (plogp) = 0, 

(i.e. the von Neumann entropy is time invariant under Hamiltonian evolution), and using 
the cyclic property of trace, we have: 

^D(pIIp) = ^trace (plogp) + ^trace (jH, p] logp + p[H, logp 

1 

= -trace ([iJ, p logp] + p[Ai^, logp]) 

= ^trace (p[Ai7, logp]) . (V.23) 

Q.E.D. 



We remark that the initial conditions for the perturbed and the unperturbed evolution can 
be different, and we can easily exchange the role of perturbed and unperturbed evolution 

12 



adding a minus sign on the right hand side. The analogy with the corresponding relative 



entropy evolution formula in classical mechanics 



is apparent. As in the classical case, 



the perturbation can be interpreted as an additive control Hamiltonian. 



B. Open Quantum Systems 



When we consider a quantum system interacting with the environment in some uncon 



34l |. the situation changes significantly. 



trollable way, namely an open quantum system 
The complete dynamical description of the situation should be done considering the tensor 
product space of both the system and the environment space. Usually, the environment 
has too many degrees of freedom to be modelled. Moreover, only partial information about 
environment initial state interactions may be available. In these cases, we can still obtain 
a dynamical equation for the system state by averaging over the environment degrees of 



freedom 



2(]. If the system evolution is assumed to be Markovian, strongly continuous in 



time and completely positive 3^, a g eneral form for the generator of the system density 



operator dynamics is the following 



(V.24) 



where H is the effective Hamiltonian, in general different from the free drift Hamiltonian, 
and the generator for the dissipative evolution C has the form: 



1 



C[p] = -j:{[L,p,Ll] + [L,,pLl 

k 



(V.25) 



The operators Lk can be derived under different assumptions on the couplings with the 
environment or on a phenomenological basis (see e.g. [2'] and reference therein). This 
equation can be seen as a quantum analogue of a Fokker-Planck equation, since it describes 
the time evolution of the density operator in the absence of conditioning measurements. 
Assume that (IV. 241) admits a stationary state commuting with the effective Hamiltonian, 
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and denote it with p. Noting that: 

/ , \ ( d \ 

trace I p— log P I = trace I —p 1=0, 

since the generator (]V.24p has zero trace, we obtain for the relative entropy production (see 
also 0): 



^D(pIIp) = trace {C[p]i\ogp - logp)) < 0. (V.26) 



The fact that D(p||p) is non- increasing for the dynamical semigroup generated by (]V.24p 
was established by Lindblad, see e.g. 4]. To extend this result to the case of a perturbed 
Hamiltonian, we consider now p as a fixed target state, since the introduction of perturbations 
could in general change the stationary states. In this setting, we get: 

d _ i ~ _ 

— ©(pIIp) = trace (-[if, p] logp + /:[p] (logp - logp)) 

= -^([Ai7,logp])^ + trace (£ [p] (log p - logp)), (V.27) 

where now p is undergoing a perturbed evolution H = H + AH and [p, H] = a.s before. In 
the quantum case, however, the effectiveness of a control Hamiltonian is severely limited. 
For instance, in the closed system case, the density operator eigenvalues cannot be modified 
by a control Hamiltonian, precluding convergence in relative entropy if the target state has 



a different spectrum from the initial condition. A detailed analysis of the dissipative case 
from a control theoretic viewpoint can be found e.g. in [sl, |41|. Whether these formulas 
could be of help in designing or analyzing control strategies will be a matter of further work 



fsee also comments on this issue in 



36 



Section VII]). 

Further analogies with the classical thermodynamics setting can be unravelled if we re- 
strict our attention to equation (1V.24P when it is derived from e.g. a weak coupling limit 
2j. This is essentially a constructive derivation of equations of the form flV.24p from the 
joint (tensor) description of the system and the environment, that is consistent with classical 
thermodynamics. In fact, the Gibbs state: 

Pa = Z-'e-^^, 
14 



where Z is the 'partition junction and H the system Hamiltonian, is a stationary state for 
the resulting equation. Since, as we aheady recalled, relative entropy with respect to the 
stationary state is non- increasing, for this class of dissipative Markovian evolutions we have 
a full correspondence with the classical mechanical case 35 1. 



VI. CONCLUSION AND OUTLOOK 

We have derived explicit dependence of the relative entropy production rate on the control 
action for various uncertain physical systems exhibiting a Markovian evolution. Further work 
is needed to find other significant applications of the results as well as possible extension to 
other, more complex, systems with Markovian evolution such as interacting particle systems. 
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APPENDIX: FINITE-ENERGY DIFFUSIONS 

Let VL :— C([io, ^i], K") denote the family of n-dimensional continuous functions, and let 
P and P be two probability distributions on Jl. The relative entropy H{P, P) of P with 
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respect to P is defined by 

HiP,P) 



Ep[\og§] if P«P 



dPi 

+00 otherwise 
Let Wx denote Wiener measure on Q starting at s G M", and let 



W:= JWx dx 



be stationary Wiener measure. Let a > 0, and denote by D be the family of distributions P 

on Q such that H{P, aW) < 00. Let JF^ and Qt denote the a-algebras of events observable up 

to time t and from time t on, respectively. It then follows from the Girsanov's theory Q] 

that P G D possesses both a forward drift (3^ and a backward drift 'y^, namely under P, 

the increments of the canonical coordinate process x{t,uj) = uj{t) admit the representations 

ft 

x{t)-x{s)= f3^{T)dT + a[w+{t) -w+{s)], to<s<t<ti, (.1) 
x{t)-x{s)= -f^{T)dT + a[w-{t)-W-{s)], to<s<t<ti. (.2) 

J s 

(3^{t) is at each time t jF^-measurable and is a standard, n-dimensional Wiener pro- 

cess. Symmetrically, 7'^(t) is ^^-measurable and w_ is another standard Wiener process. 
Moreover, (5^ and 7^ satisfy the finite-energy condition 

eJ^*' ■ < 00, E 1^*' 7^(t) ■ 7-^(t)dt| < 00. (.3) 

It was shown in [13] that the one-time probability density Pt{') of ^{t) (which exists for every 
t) is absolutely continuous on R*^ and the following relation holds a.s. Vt > 

E{p''{t)-^''{t)\x{t)} = a'\/\ogMx{t)). (.4) 

Let us introduce the current drift and the current drift field of P 

v^{t) = ^^^^^^, v^{x,t) = E{v^{t)\x{t)=x}. (.5) 

Then, the one-time density pt satisfies weakly [32] a continuity type equation 

^ + V ■ (v^pt) = 0. (.6) 

Hence, Theorem IIII.ll holds true for finite energy diffusions provided we define the v fields 
according to (TSi) . 
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